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(54) Title: CAMERA WITH OBJECT RECOGNITION/DATA OUTPUT 
(57) Abstract 

A line scan digital camera (12) is di- 
rected at a station for recording and displaying a 
time-sequential scene. The digital camera (12) 
takes a sequence of digital image frames repre- 
sentative of one or more bodies crossing a plane 
(FL) in space, wherein each frame represents a 
line image of the body, thus forming a fractional 
part of the scene, and the set of time-successive 
frames forms a recognizable image on an exter- 
nal display (42). In a preferred embodiment, the 
camera (12) is an event camera at a competition, 
and it images the finish line (FL) or an interme- 
diated station (L) to determine crossing times of 
contestants. The camera (12) is also adapted to 
robotic vision applications for stations at irreg- 
ular transport lines, and for mid-course identifi- 
cation and reporting along race tracks, transport 
lines and traffic environments. 
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CAMERA WITH OBJECT RECOGNITION/DATA OUTPUT 

Technical Field 

5 The present invention relates generally to systems which monitor and 

record motion events, and it relates to cameras and systems for time-sequential imaging 
and display, with application in numerous fields. Most particularly, the invention 
provides a station imaging camera and system useful for measuring timed sporting 
events and imaging movement along defined tracks at stations. The invention also 
1 0 relates to systems and methods for generating a scene by compiling successively- 
scanned line objects, as described for example in applicant's commonly-owned United 
States patents 5,552,824 and 5,657,077. 

Prior art systems employing standard photographic techniques to monitor 
1 5 the finish line of a race are known. In such a system, typically one or more cameras 

equipped for high resolution imaging view the finish line and capture sequential pictures 
at a high rate for later inspection by a judge or other interpreter. However, this process is 
cumbersome, wasteful, and time-consuming, in that it requires, for example, an 
apparatus of photographic film and paper, processing chemicals, and image enlargers or 
20 projection optics to be employed with their respective methods of operation, 

development and finishing. Consequently, most races rely on human judges and revert 
to "photo-finish" technology only in extremely close or important events. The Specialty 
Instrument Corporation provides a number of electronic and photo-finish systems of this 
type marketed under the trademark Accutrack. U.S. Patent No. 3,829,869 exemplifies 
25 one such Accutrack system. 

Because of the problems with the "photo-finish" technology, numerous 
other systems for monitoring racing events have been developed. However, these other 
methods and systems for timing sporting events present new difficulties. Video systems 

30 which record and display races in a standard television or video format are popular, but 
regardless of the particular implementation of these systems, a portion of the electronic 
image remains on an analog medium, such as recording tape. Since analog data from 
the systems consists of a continuum of information over time, it is relatively difficult to 
accurately apportion to a unique time interval. It is even more difficult to access a 

35 particular moment in time in the recorded sequence because the associated system must 
search the storage medium, typically having a long physical length in a spooled format, 
e.g., a video cassette. This presents both limitations and difficulties for users wishing to 
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linear/temporal or t,y-dimension image may be formed that bears a readily interpretable 
similarity to the customary optical spatial or x,y-dimension image of the scene. As set 
forth in applicant's above mentioned patents, the data stream from such cameras can be 
used to detect and deal with moving objects to provide high temporal and spatial 
5 resolution in real time. This entails transmission of a generally continuous stream of line 
image data to a processing system, which then attends to the annotation, indexing, 
compression and storage of the relevant views so that a small sub-portion of relevant 
views can be placed in digital random access storage and readily recalled, typically 
within minutes or seconds of the original image acquisition, for detailed inspection. 
1 0 However, effective use of such line-imaging camera systems has required extensive 

software-mediated data handling by a trained technician operating the system, and great 
demands are imposed on the data transmission and synchronization for effecting image 
assembly, time synchronization, and image frame recording and access. 

15 It is, accordingly, an object of the invention to provide an improved 

camera and system for recording and displaying a time-sequential scene of bodies 
crossing a plane along a track. 



20 



These and other objects will become apparent in the description below. 
Summary of the Invention 



The invention features, in one aspect, a camera for forming a time 
sequential scene of bodies moving across a plane in space, wherein the camera 

25 recognizes the appearance of an object within its limited field to trigger or flag its image 
output stream. The system includes at least one camera which is aimed to image objects 
crossing a line of interest, wherein the camera time-sequentially captures the object by 
imaging it onto an array of detector elements and converts the sampled signal into a 
digital line image, or frame, which passes to a frame buffer. Each digital image frame 

30 uniquely represents a fixed slice of the moving scene at a moment in time. A processor 
is situated within the camera, and communicates with the buffer, processing information 
from corresponding pixels or larger blocks of time offset frames to detect an object 
which has entered the line field of view and responsive thereto, controls the image data 
output stream or produces data coordinated with the image stream. For example, in one 

35 aspect, the camera detects arrival or departure of objects in the image field and 

thereupon operates to produce or enable an image data output stream, or to annotate the 
stream and enhance its information content by indicating such detection. In a basic 
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features in the correct order before triggering the finish line time determination, or by 
otherwise confirming the presence of an object, the camera avoids falsely triggering, for 
example on a shadow of a back-lit runner that may precede the actual crossing, 

5 In yet another or further embodiment, the processor processes frame data 

to detect patterns or markings on the imaged objects, such as numbers identifying the 
contestants, and the output frames in which these objects or indicia were detected are 
flagged or indexed with the detected identifiers. The line camera may be set up at a 
plane of interest, such as the finish line of a race or an intermediate position along the 

10 track. In this case, the frame times may also be processed with the flagged identifiers at 
the observed plane, either the finish line or an intermediate position, to quickly show 
rankings or relative positions, speed or other ratings for immediate display in an 
automated manner without waiting for a judged inspection of the digitally reconstructed 
images as required in previous image timing systems. The camera output augmented 

1 5 with this data thus directly interfaces with a computerized race management system so 
that desired frames are quickly accessed by their index information for review. A user 
at the computer console can command a variety of functions provided by the invention 
to manipulate and analyze the captured scene, most particularly to display any portion of 
the scene of bodies moving across the plane and access an associated time and/or 

20 identities of contestants for each frame within the sequence of line crossing images. 

The system of the invention is particularly useful in recording and 
managing the official times of objects or entrants crossing a finish line in a digital race 
management system since it both preserves all imaging information and simplifies the 

25 storage, transmission and selective retrieval of finish line information. A user can 

therefore record and display the bodies crossing the plane of interest, i.e., the finish line, 
with accuracy while enhancing the ability to immediately employ information embedded 
in the images in real time, both to review and edit the stored images for finish 
determinations, and to effect fast automated display of unreviewed results. The 

30 invention thus provides an object-controlled compression and information enhancement 
of the camera output data. 

The system constructed in accordance with the invention also provides, in 
another aspect, an in-camera memory subsystem communicating with an in-camera 
35 processor, wherein blocks of frame data are processed to detect an object, to detect a 

background, and to control start time and/or end time for active imaging segments which 
are to be transmitted out of the camera for analysis. According to this aspect of the 
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FIGURE 1 illustrates a camera 12 in accordance with the invention 
mounted in a system for imaging and recording a sequence of bodies crossing a plane in 
space, which illustratively is shown containing the finish line FL of a track. As 
discussed further below in connection with certain embodiments, a camera 12a may also 
5 be positioned to image an intermediate line L in a race monitoring system. In other 
aspects the system may be positioned to image a station in a robotic process line; 
however, for clarity of exposition below, the example of an event recording camera 12 is 
discussed in detail. 

10 As shown in FIGURE 1, the camera 12 is mounted on a stand S which 

places the camera above and to the side of the finish line, with the camera itself oriented 
and aligned such that the finish line is imaged onto a linear sensing array, the output of 
which is electrically scanned or sampled at a high rate to produce a time-resolved image 
data stream. Preferably the camera is positioned in the vertical plane of the line FL, and 

1 5 aligned so that the plane is projected onto the sensing array. For example the camera 

may be above and to the side of the track, or in certain cases may be mounted overhead. 
The camera is connected in a race management system 1 0 which includes the digital 
camera 12 and an image management computer 16 connected to various display or 
recording devices and interfaces. 

20 

As shown more fully in FIGURE 1 A, the camera 12 of the present 
invention possesses an objective lens or optical assembly 15 which images a line portion 
18 of the object field of its objective lens 15 onto an array of detector elements 20, 
preferably a Line Scan Charge Coupled Device (LS-CCD) having a pixel length of one 

25 thousand to several thousand pixels. An imaging controller 22 samples the detected 

light level at the detector elements 20 at successive times and amplifies the output signal 
with a gain controlled detector amplifier 24 and digitizes the processed signal with an 
A/D converter 26, to acquire successive frames of image data. The signal frames, each 
representing a line of pixels, are taken at a rate above several hundred lines per second, 

30 preferably about 400-1000 lines per second, and are stored in a buffer 25. Buffer 25 
may, for example, be implemented with about one megabyte of VRAM, corresponding 
to somewhat under one minute of raw image frame data at that sampling frame rate. 
Each sampled image represents a frame of digital information at a unique moment in 
time, and may illustratively have the format of an ordered data string with a pixel value 

35 for each of a thousand successive pixels forming the line array. As set forth in greater 
detail below, the frames from the sensor and residing in the buffer are further processed 
in the camera to form the camera output stream. 
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the frames acquired in the buffer 25 to detect object information in the image, and, 
responsive thereto, annotate or control the image data output on the output signal line 28 
for connection to an external image management system 1 6 It will be understood that 
line 28 is shown for purposes of illustration as a line, but is more generally to be 
5 understood to be a communication link, such as a local radio frequency link so that the 
camera may alternatively send its data output by RF transmission. 

Several representative object detection operations will now be 
illustratively presented. It will be familiar to those in the art that two-dimensional (line 

10 jc time) image frames created by an event recording line camera appear smeared in the 
width direction, with the jc- or width dimension proportional to the dwell time of the 
object at the imaged object line L or FL, and inversely proportional to the speed of the 
object crossing that line. Conversely, background field, when not obscured by a 
contestant passing in front of it, remains always the same, forming horizontal bands of 

15 fixed color or intensity passing through each pixel position of the line sensor 20. These 
are the image frames in buffer 25 upon which the object detector described below 
operates. Preferably each line image frame is marked with a time reference, so that this 
time information appears within the digital information of the frame. In a preferred 
embodiment, the time reference for each frame is indicative of the time the camera 12 

20 captured the picture. This may be an absolute time, or a time relative to the start of an 
external event or trigger signal, and it may be derived from other signals, such as the 
signal from a start detection sensor 84 (FIGURE 1), or an external time source, or an 
internal camera timer, preferably with a system time synchronization protocol as 
described in applicant's aforesaid patents. 

25 

In general, the external computer storage and image analysis system 16 
may include elements such as software for adding text or linking images with 
identifying data to index the raw image data, and for indexing, accessing, and quickly 
displaying and analyzing the times appearing in the image data. Systems of this type are 
30 extensively described in the aforesaid two commonly owned U.S. Patents. 

By way of context, the main control computer 1 6 of the external 
management system may have a central processor 40 that processes the blocks of 
information stored within an internal memory 36 and organizes the line frames into two- 
35 dimensional image frames for displaying the scene and time contents of a sequence of 
digital line frames on a display monitor 42. The central processor 40 preferably also 
controls the automatic operation and memory management of the system 10, and 
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line, such as a wall, is imaged identically in successive line images, so that when a frame 
is assembled from successive lines, the background appears as horizontal stripes each 
having the fixed color or intensity of the imaged stationary feature. 

5 FIGURE 3 illustrates a typical image frame formed in this fashion of a 

group of runners crossing a finish line. Among the artifacts due to the mode of 
generation of this image,, it is apparent that the background appears as a set of horizontal 
stripes when not occluded by a runner, and each runner appears frozen in the exact 
posture he assumed as successive strips of his body crossed the finish line. Nonetheless, 
10 except for a slight dimensional stretching or compression effect along the lateral 

direction, each runner appears approximately as in a normal spatial image taken from 
the perspective of the fixed camera position. Thus his suit color, entry number, facial 
traits and other identifying features are, or would be, captured in the image. 

1 5 Returning now to a description of camera operation, each digital line 

frame captured by the system 10 of FIGURE 1 A is stored in the buffer 25, and the 
control microprocessor 22 operates on this line image data to control the camera output. 
The invention contemplates several different forms of object recognition and output 
control, examples of which will now be described. First among these is the recognition 

20 of an object moving across the line FL. 

FIGURE 4 is a flow chart illustrating this operation in a basic prototype 
embodiment of the invention. In this embodiment, the microprocessor 22 in the camera 
processes raw image frames from the buffer 25 to detect the presence of an object 

25 moving across the line field of view, and it controls the output data stream so that the 
image frames are sent out by the camera only when there is finish line activity, while 
precision time marking is maintained. In terms of hardware, the microprocessor loads 
the line frames into VRAM of moderate size, e.g., a 2MB VRAM, and then processes 
successive lines to detect finish line activity. In a further aspect, when a pattern of 

30 probable motion is detected, the processor confirms the pattern, and retroactively tags 
the start of the active image sequence for transmission out of the camera, while inactive 
frames or "dead time 11 images are simply compressed, deleted, or even sent to tape 
backup without occupying system bandwidth. 

35 In addition to the buffer storage or VRAM, the camera's internal 

microprocessor 22 preferably has a RAM capacity of about 32MB or more, allowing the 
camera itself to store up to several minutes of time-resolved active finish line images. 
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time tg of the first line of the current block. In alternative embodiments, the processor 
may wait for two, or more generally n blocks in a row to differ, so that it confirms the 
sustained presence of an object before it annotates the image data or sets an output 
control signal. In that case the camera starts counting, and when the block value 
5 remains different and above the threshold for at least, e.g., five consecutive blocks in 
time, the change is recognized as being due to an object moving across the field of view. 
The processor retroactively sets the BEGIN-OBJECT flag at the start of the active 
sequence. The flag data may be sent as a separate data indication along with the 
continuous image data stream to facilitate external processing and indexing of the line 

1 0 image data, or the flag data may be used to directly initiate the output of image data 
from the camera commencing at time tQ, so that image data is sent to the output port 
only when an object has been detected. In another embodiment, the camera may be set 
to catch the image data starting at a preset time before the first detected change frames, 
for example at . 1 seconds before the first detected change, so that the finish line 

1 5 sequence will include images as the contestant nears the finish and shows the first parts 
of a contestant crossing. Thus, the nose of a horse, or hand of a runner, will appear 
before the torso breaks the ribbon. Such details, while coming earlier than the event 
which is traditionally recognized as crossing the line, are useful for resolving 
ambiguities when multiple contestants cross nearly simultaneously. For example, a 

20 projecting hand may be followed to identify the position of a contestant even when the 
torsos of several contestants are simultaneously occluded or overlapping in the field of 
view. Thus, the active image sequence may be set to include a fixed leader portion. 

As further shown in FIGURE 4, once the runners have been detected in 
25 the image the processor then switches to a different analysis procedure which monitors 
or analyzes the object now appearing in the image frames to derive additional 
information. In a basic embodiment, the processor continues processing sample image 
strips in a manner somewhat analogous to the initial object-trigger analysis, to determine 
when the object has passed or the finish line activity ceases. A suitable set of 
30 determinations is shown in FIGURE 4A for this aspect of the processing. When the end 
of the object is detected, the processor then sets an END OF OBJKCT flag, preferably 
causing the output controller to turn off the output stream or enter a dormant mode. For 
this latter detection, the processor detects whether all pairs of blocks differ by less than a 
threshold Tj- Advantageously, the processor may employ a different threshold, T2 < T\ 
35 or may require that a sub-threshold level of homogeneity persist for a greater number of 
sample strips, or both, before concluding that the moving objects have passed. This 
operation avoids falsely signaling the end of a sequence when in fact a relatively 
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In addition to flagging the active segment of the image data generated' by 
the camera'? line sensor, the invention contemplates further internal processing of 
identified object to produce object-based data. This is shown schematically in 
FIGURE 5. Two distinct types of data will be briefly illustrated. One is object 
5 identification, which proceeds by way of image analysis subroutines to detect color, 
number, bar code or other individual features of the contestants as they pass. This is 
discussed in greater detail below. The second, which will now be discussed, may be 
called "photocell determinations". In accordance with this latter aspect of the invention, 
the in-camera processor performs a more detailed image analysis of the object- 
1 0 containing frames to fit the contestant to a template, recognizing defining parts of the 
body, and then applies a species/event specific interpolation or estimation rule to 
determine the probable crossing time. 

As applied, for example to a horse, the processor may process direct- 

1 5 overhead image lines to detect the body of a horse crossing the line, then, while 

correcting for frame rate and event speed (which affects the horizontal distance scale), 
identify the crossing time by the frame which occurred a fixed time or distance before 
the mid-point of the body. This estimation procedure will produce an estimated time, 
based on a standard species size and shape, which can be used for initial announcement 

20 scoreboard displays. For example, the processor may set the "nose" of a horse to be 1.6 
meters ahead of its center, or the nose of a greyhound to be .7 meters ahead of its center. 
By adding the identified time to the image output data stream, an immediate index is 
provided to the race management system, and full-frame images may be immediately 
constructed and reviewed to determine the exact crossing time. Inspection of FIGURE 3 

25 shows a typical crossing configuration for a human track event. As applied to a person, 
the relevant image analysis may be triggered by the initial appearance of a hand, arm or 
leg in the field. In that case, the image analysis may proceed by a fixed rule, such as 
setting the probable crossing to be about 30-50 centimeters after the initial limb, or it 
may proceed with further image analysis to detect the central mass of the body, e.g., the 

30 torso or stomach, and to set the estimated crossing time at the appearance of this 

feature. A race official would then view the image frames so identified to determine the 
exact crossing time by observing, for example, whether the torso were angled forward, 
and seeing which line of the composite frame actually contained the leading edge of the 
torso. Other photocell crossing determinations may be readily implemented by 

35 employing rules specific to the speed of the event and the species of contestants. 
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colors at once. The invention also contemplates systems wherein a special identification 
marker is worn by each contestant so as to be visible in the image. Thus, for example, 
when used for an equestrian event, the markers may be attached to the top of each 
horse's back, and the camera may image the finish line (or an intermediate station along 
5 the track) from directly overhead so that all markers are oriented directly toward the 
camera and no marker is ever blocked by the adjacent runners. Such markers may 
include, for example, oriented bar codes, or lines or blocks of different colors forming a 
distinctive combination. The color markings may also be advantageously shaped to 
assure quick and dependable detection. This may be done, for example by arranging the 

10 different colors in color bands which are oriented in a manner to assure that they are 
imaged on separate detection pixels of the camera or have a substantial dwell time on 
the detecting pixels. For example, the bands may be worn parallel to the direction of 
motion, giving them an extended dwell time in the image, and be offset across the 
direction of travel so that they are imaged onto different pixels of the sensor. They may 

1 5 also, where size permits, be offset along the direction of travel, leading to temporal 
resolution of the color combination in the image. This makes them readily detected 
when the line images are sampled in blocks of narrow height along the image sensor or 
are imaged for only a short time duration. 

20 In yet other embodiments, the processor may incorporate an optical 

character recognition module to simply read the number on a contestant's jersey, or may 
process the image lines using patterns of a bar code reader to recognize bar code 
identifiers. In general, however, the use of a mark configured for line-camera detection, 
and preferably color identification markers is preferred to the use of character or bar 

25 code technology, in order to reduce the possibility of occlusion and misreading of the 
sought-for markers. 

When used as an event recording system, such as at the finish line of a 
race, the object triggered line scan camera system of the present invention may be 

30 operated to record and resolve positions one thousandth of a second apart, or less, while 
greatly reducing the real time bandwidth demands imposed on external system elements, 
and without introducing any noticeable delay in the transmission of necessary image 
data. The data annotation of output frames, even without physical cropping of static or 
dead time images, also results in a system wherein the data handling and RAM storage 

35 demands placed on the external race management system are reduced, 
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Claims 

1 . An event recording camera system for forming a time sequence of 
images, comprising 

a camera which generates a first sequence of digital line image frames of 
a fixed position in space, each frame of said first sequence representing a line image 
view that is captured by said camera at a moment in time and imaged onto an array of 
detector elements, said first sequence including active segments imaged at times 
wherein moving objects cross said fixed position, and inactive segments wherein no 
moving objects cross said fixed position 

a buffer in said camera for storing said first sequence of frames as they 
are generated, and 
15 

a processor in said camera operative on said first sequence to detect an 
object and produce an object detection signal so as to selectively present said active 
segments at a camera output. 

20 2. A system according to claim I , wherein said processor 

communicates with the buffer to receive data from frames of said first sequence and 
compare data from successive frames to detect an object, such that said processor 
retroactively identifies start of an active imaging segment. 

25 3. A system according to claim 1, wherein frames of the active 

imaging segment include frame time information. 

4. A system according to claim 1 , wherein said processor 
communicates with the buffer to receive data from frames of said first sequence and 

30 compare data from successive frames to detect end of an object, and sets the end of an 
active imaging segment. 

5. A system according to claim 1, wherein the active imaging 
segment includes object identification information. 

35 

6. An event recording digital camera including an optical imaging 
element for forming a light image of a restricted visual field, and an image sensing 



5 
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13. A digital camera according to claim 12, wherein the processor 
annotates the image data stream with data codes indicative of at least one of start of 
object, end of object, and identity of object. 

14. A digital camera according to claim 12, wherein the processor 
controls output of the camera to selectively pass image frames containing moving object 
images. 

15. An event recording video camera comprising 



means for imaging a visual field 

a line sensor for converting a line portion of the imaged visual 
field into electrical signals representative of pixels of the image of said field 

circuit means for processing said electrical signals to form a 
15 digital output stream containing digitized pixel values of a sequence of frames at a high 
scanning rate 

a buffer for storing said digital output stream before transmission, 

and 

a processor operative on contents of the buffer for comparing sets 
20 of data from the buffer and identifying image data associated with moving objects, said 
processor controlling said output stream responsive to the identified data. 

16. A video camera according to claim 15, wherein said processor 
compares block sum light values to detect beginning or end of object motion in the 

25 visual field. 

17. A video camera according to claim 15, wherein said processor 
image-processes blocks of frames to detect an object identifier and controls said camera 
to transmit image data annotated with an indication of object identifier. 



18. A video camera according to claim 15, wherein said processor 
image-processes blocks of frames to detect an object feature and controls said camera to 
transmit image data and estimated crossing time data. 



35 



19. A video camera according to claim 1 5, wherein said pixel values 
include color values and said processor processes values for each color separately. 
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